To recap, XGBoost stands for Extreme Gradient Boosting and is a supervised learning algorithm that falls under the gradient-boosted decision tree (GBDT) family of machine learning algorithms. They make their predictions based on combining a set of weaker models and evaluate other decision trees through if-then-else true/false feature questions. They are created in sequential form to assess and estimate the probability of producing a correct decision. Before we get into the tuning of XGBoost hyperparamters, let's understand why tuning is important Hyperparameter tuning is a vital part of improving the overall behavior and performance of a machine learning model. It is a type of parameter that is set before the learning process and happens outside of the model.

Before we get into code and muddy our hands, let us hold here for a minute and ask ourselves- if we were a computer and had been given the same problem, how would we do it? I'll assume we have only two hyperparameter values in this situation because it makes it easier to visualize. The first thing I am going to do is built a model with any random values for our two hyperparameters and see how my model performed. The next thing I would do is increase one parameter and keep one stationary just to see how my model performance responds to an increase in one of these parameters. If my model performance increases, that means I am moving my hyperparameter in the right direction.